We explore combining dropout with robust training methods and obtain better generalization.

Current methods for training robust networks lead to a drop in test accuracy, which has led prior works to posit that a robustness-accuracy tradeoff may be inevitable in deep learning. We take a closer look at this phenomenon and first show that real image datasets are actually separated. With this property in mind, we then prove that robustness and accuracy should both be achievable for benchmark datasets through locally Lipschitz functions, and hence, there should be no inherent tradeoff between robustness and accuracy. Through extensive experiments with robustness methods, we argue that the gap between theory and practice arises from two limitations of current methods: either they fail to impose local Lipschitzness or they are insufficiently generalized. We explore combining dropout with robust training methods and obtain better generalization. We conclude that achieving robustness and accuracy in practice may require using methods that impose local Lipschitzness and augmenting them with deep learning generalization techniques.

Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. Moreover, the training process is heavy and hence it becomes impractical to thoroughly explore the trade-off between accuracy and robustness. This paper asks this new question: how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies, without (re-)training it many times? Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework, with a controlling hyper-parameter as the input. The trained model could be adjusted among different standard and robust accuracies "for free" at testing time. As an important knob, we exploit dual batch normalization to separate standard and adversarial feature statistics, so that they can be learned in one model without degrading performance. We further extend OAT to a Once-for-all Adversarial Training and Slimming (OATS) framework, that allows for the joint trade-off among accuracy, robustness and runtime efficiency. Experiments show that, without any re-training nor ensembling, OAT/OATS achieve similar or even superior performance compared to dedicatedly trained models at various configurations.

Szegedy et al. [1] were the first to demonstrate that small, imperceptible perturbations to input data can lead neural networks to make incorrect predictions with high confidence. This discovery exposed a significant vulnerability in machine learning models and introduced the concept of adversarial attacks. In recent years, the vulnerability of deep learning models to adversarial attacks has driven significant research into improving model robustness [1, 2]. Adversarial training, widely regarded as the most prominent defense against adversarial machine learning (AML) attacks, enhances model robustness by incorporating both benign and adversarial examples into the training process [3]. However, it often leads to reduced accuracy on clean data [4].

We explore combining dropout with robust training methods and obtain better generalization.

We thank the reviewers for their encouraging and instructive comments, and the AC for guiding the review process. Gray (2013), and may look a bit too complicated. We will add a remark in line with our comment above. Note that the assumption on encoder gap is very mild. R2: It is not clear that sparsity-promoting encoders are the right models to be studying. Ours is the first work to address this.

We explore combining dropout with robust training methods and obtain better generalization.

We thank all reviewers for their comments. We are glad everyone found out paper well written. Below we address specific comments. "There exist no real world data where the classes are well-separated... images suffer from different lighting and To be concrete, here is a turtle and a fish from Restricted ImageNet. Moving 2 r from the turtle to the fish still looks much more like a turtle. "The classifier proposed in the existence proof essentially computes the distance of a test point to every point in the However, our theory result is not intended to be used in practice.